Bridge: A High-Performance File System for Parallel Processors

Faster storage devices cannot solve the I/O bottleneck problem for large multiprocessor systems if data passes through a file system on a single processor. Implementing the file system as a parallel program can significantly improve performance. Selectively revealing this parallel structure to utility programs can produce additional improvements, particularly on machines in which interprocessor communication is slow compared to aggregate I/O bandwidth. We have designed and prototyped a parallel file system that distributes each file across multiple storage devices and processors. Naive programs are able to access files just as they would with a conventional file system, while more sophisticated programs may export pieces of their code to the processors managing the data, for optimum performance. Theoretical and early empirical data indicate nearly linear speedup on critical operations for up to several dozen devices.